Farming in Data Rich Environment
Jianqing Fan, Department of Operations Research and Financial Engineering, Princeton University, USA
Correlated and heavy-tailed data arise frequently in a wide range of scientific and engineering problems: from genomics, medical imaging to neuroscience and finance. This talk introduces Factor-Adjusted Robust Multiple testing (FARM-test) and Factor-Adjusted Robust Model selection (FARM-select). The former is introduced to control the false discovery proportion for large-scale simultaneous inference when variables are highly correlated, and the latter deals with variable selection problems when covariates are highly correlated. We demonstrate that robust factor adjustments are extremely important in both improving the power of the tests and controlling FDP. We identify general conditions under which the proposed method produces a consistent estimate of the FDP. We also prove that factor adjustments significantly reduce the conditions needed for selection consistency. The results will be illustrated by numerical experiments. This is joint work with Yuan Ke, Qiang Sun, Wenxin Zhou, and Kaizheng Wang.